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Executive Summary 


The District of Columbia Opportunity Scholarship Program (OSP) continues to be a focal point in debates about 
school choice policies. Aimed at improving academic performance in the nations capital, the program provides 
tuition vouchers to low-income parents so that their children can attend private instead of public schools. While 
a few other cities and states also have voucher or private school scholarship programs, the OSP is the only one 
created and funded by the U.S. Congress. 

This report — the first from a new evaluation of the program — focuses on how the OSP has been implemented 
in the 2 years since it was reauthorized under the Scholarships for Opportunity and Results (SOAR) Act of 
2011. Because the 2011 law expanded the scholarship amount, the types of students who receive priority in 
receiving scholarships, and the accountability requirements for private schools, it could lead to changes in the 
mix of program participants. For that reason, the report not only describes the current status of the OSP under 
the SOAR Act but also takes a look back to the earlier years of the program since 2004 in order to provide 
some perspective on how it is now operating. Information for this analysis was drawn from a variety of sources, 
including records from the OSP program operator, federal databases, school websites, and published reports 
prepared under a prior evaluation of the OSP. 

The report explores three questions about the schools, the applicants, and the award and use of OSP scholarships. 

Private Schools: How many participate, and what are their characteristics? 

School choice theory predicts that successful voucher programs will offer prospective parents an attractive set of 
school options and at the same time offer private schools an attractive payment and set of procedures. Examining 
the private schools that participate in the OSP and how they compare with other private schools and public 
schools suggests four key findings: 

• Just over half of all DC private schools participate, with a smaller core set involved since the program 
began. The number of participating schools has shifted over time, from 57 in the first year the program was 
operating (2004-05) to a high of 68 the second year, to 52 in the most recent year under the SOAR Act 
(2012-13), with 33 schools involved in the program continuously since 2004-05 (figure ES-1). Exits from the 
program over time include nine schools that converted to public charter schools and could no longer receive 
OSP vouchers, four that shut their doors, and five that continued operating but withdrew from the program 
(figure ES-2). Across the years, participating schools have represented at least half of all private schools operating 
in DC. But after the first year, when 57 of 109 DC private schools (52 percent) signed on to the OSP, the share 
declined from 65 percent (2005-06) to 54 percent (2012-13). These trends suggest that the changes in the 
2011 law have not drawn more private schools into the OSP. 

• Participating schools are more likely now than in the past to report tuition rates above the OSP scholar- 
ship amounts, to have no religious affiliation, to serve grades 9-12, and to have less diverse student popula- 
tions. The characteristics of the pool of private schools now available to OSP families differ from the earlier years 
of the program, comparing years for which comparable data are available (figure ES-3). Proportionally more 
schools participating under the SOAR Act in 201 1-12 than in 2005-06 include high school grades (36 percent 
versus 22 percent). Proportionally fewer are affiliated with a religious denomination (62 percent versus 68 
percent previously). Recent OSP private schools serve a smaller percentage of students from minority racial- 
ethnic groups than did schools participating in earlier years. Most important, a larger proportion of participating 
schools under the SOAR Act have published tuition rates that are above the legislated scholarship amount, while 
relatively few did under the earlier DC Choice Incentive Act ( e.g ;, 64 percent in 201 1-12 versus 39 percent in 
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Figure ES-1 Total Number of DC Private Schools and Private Schools Participating in the OSF? 
2004-05 Through 2012-13 



04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13 


Years Schools Participated 


Total Non-Participating 
Private Schools 

Total Participating 
Private Schools 


NOTE: *2009-10 and 2010-11 were after the prior evaluation stopped collecting data. There are no reliable historic records of how many private 
schools existed during those years. Years refer to school years. The total number of private schools operating in DC each year is represented 
by the number at the top of each bar, with the bar divided into the number not participating in the OSP (top, blue shading) and the number 
participating (bottom, grey shading). So, for example, in 2004-05, out of 109 private schools in DC, 52 did not participate in the OSf? while 
57 participated. 

SOURCES: Data for the total number of private schools are from Evaluation of the DC Opportunity Scholarship Program: Final Report 
(NCEE 2010-4018), the Private School Universe Survey (PSS) ( http://nces.ed.gov/surveys/pss/pssdata.asp) , and private school websites. 

Data for participating schools are from the Trust’s Participating School File. 


Figure ES-2 Number of Private Schools Leaving, Joining, and Continuing in the OSF? 
2004-05 Through 2012-13 



NOTE: Years refer to school years. 

SOURCE: Data for 2004-05 through 2012-13 are from the Trust’s Participating School File. 
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2005-06. The extent to which families pay the difference will be examined in future evaluation reports, taking 
into account patterns suggesting that OSP students cluster in participating schools that do not charge tuition 
above the voucher cap. 

• Private schools that currently participate in the OSP have been operating longer, are more likely to be 
religiously affiliated, and have larger class sizes than other private schools in DC. The DC private school 
sector is diverse, with schools that vary in their selectivity, target populations, affiliation, and other characteristics. 
On average, participating schools have been in operation for 75 years and have a pupil- teacher ratio (a proxy for 
class size) of 9, compared to 50 years and 7 for those not participating in the program. A higher share of partici- 
pating schools than non-participating private schools is religiously affiliated (64 percent versus 29 percent). 

• Compared to the public schools parents may be considering, participating schools are smaller, serve a 
higher share of White students, and are clustered in affluent areas of the city. Parents of eligible applicants, 
if they receive a scholarship, have choices from among participating private schools and a wide variety of public 
schools, including charter and magnet schools spread across the district. These parents may view the charac- 
teristics of private schools as “proxies” for school quality because test scores cannot be compared across sectors. 
Participating private schools enroll fewer students (average of 243 versus 348) and have lower pupil-teacher 
ratios (“class size”) than traditional public and charter schools (on average 9 students per teacher versus 12). 
Participating schools also have a higher proportion of White students than public schools (on average 35 percent 
versus 6 percent). About 6 out of 10 OSP schools (57 percent) are located in the four most affluent sections of 
the city, in wards (clusters of neighborhoods) with average annual household income above $100,000 (Wards 2, 
3, 4 and 6). 

Applicants: What is the nature of demand for the program among eligible students 
and families? 

The 2011 re-authorization significantly increased funds for the program (from $14 million per year under the 
earlier statute to $20 million per year) in order to expand the availability of scholarships. The main available 
indicator of family interest in the OSP is applications to the program. But because the number and type of eligible 
applicants also reflect the decisions and efforts of the program operator and of the U.S. Department of Educa- 
tion, applications are best considered measures of “generated” demand, while the true demand for the program is 
unknown. 1 Exploring the nature of eligible OSP applications across years and the factors that may contribute to 
changes in demand suggests four key findings: 

• The number of applications taken has fluctuated, mosdy along with funds available to admit new 
students. Since the OSP began, the number of applications from eligible students has varied (figure ES-4). 

To some extent, the number of annual applications appears to be linked to key milestones in program opera- 
tions. More students and families applied in the years immediately after the two congressional authorizations 
(2003 and 2011), when scholarship support for new applicants was most available. In contrast, in other years 
(2006-07 through 2008-09), the program operator scaled back recruitment because OSP funds were primarily 
used to support continuing students, with the remaining resources sufficient only to replace students leaving the 
program. 2 No applications were taken in 2009—10 and 2010—1 1 because Congress had closed the program to 
new students and provided funds only to support continuing scholarship recipients. However, despite the greater 
funding available under the first two years of the SOAR Act, there were fewer eligible applicants than during the 
comparable period of the earlier statute. There are no data available to explain these differences, though possible 
hypotheses include changes in program recruitment practices or the increase in public school choice options 
available to parents in DC over this period. 


1 For the purpose of this report’s analysis, for example, eligible applicants are only those students eligible to receive a scholarship in a given year. For example, 
in some years, otherwise eligible students who were already attending private schools when they applied to the OSP were not permitted to enter lotteries to 
determine the award of scholarships because the legislation allows for targeting of OSP funds to families that cannot exercise school choice. 

2 See p. 26 of a report prepared by the Government Accountability Office on the OSP under the DC School Choice Incentive Act, available at http://www.gao. 
gov/products/GA0-08-9 . 
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Figure ES-3 Characteristics of Participating Private Schools Under the DC School Choice Incentive Act and 
the SOAR Act, 2005-06 and 2011-12 


100 


<« 



Percent Serving Percent Reporting Average Percent Percent Religiously Percent Catholic 
Grades 9-12 Tuition Rates Minority Students Affiliated 


Above OSP Cap Served* 

School Characteristic 


Incentive Act 2005-06 


SOAR Act 2011-12 


*“Average Percent Minority Students Served” is calculated as the mean of the percentage of each school’s student body that is from a minority 
racial-ethnic group (non-White). 

NOTE: The 2011-12 school year (under the SOAR Act) and the 2005-06 school year (under the DC School Choice Incentive Act) are used to 
compare the characteristics of the program under the two statutes, with these specific years chosen because: (1) both time periods were soon 
after each statute was passed and when significant funds were available for new scholarships, and (2) comprehensive data from the earlier 
evaluation were available for the 2005-06 year that were not available for other years. Tuition data for participating schools in 2005-06 are 
available for 60 of the 68 schools and for all 53 participating schools in 2011-12. Years refer to school years. 

SOURCES: Data for 2005-06 come from Evaluation of the DC Opportunity Scholarship Program: Second Year Report on Participation (NCEE 2006- 
4003); 2011-12 data are from the Trust’s OSP Participating School Directory and private school websites. 


Figure ES-4 Number of Eligible Applicants, 2004-05 through 2012-13 
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SOURCES: Data for 2004-05 through 2008-09 are from Evaluation of the DC Opportunity Scholarship Program: Final Report 
(NCEE 2010-4018); data for 2011-12 and 2012-13 are from the Trust’s OSP Applicant database. Years refer to school years. 
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• Under the SOAR Act, OSP applicants represent fewer than 5 percent of eligible DC students. The law sets 
clear residency and family-income criteria to determine student eligibility. Students must be residents of 
DC and have an annual household income within 185 percent of the 2011 federal poverty level ($34,281 for a 
family of three and $4 1,348 for a family of four). The OSP operator accepts free and reduced-price lunch eligi- 
bility as proof of meeting this income criterion. The 1,550 OSP applicants in the first 2 years under the SOAR 
Act (see figure ES-4) represent between 3 and 4 percent of the estimated 53,000 children in DC who meet the 
eligibility criterion. 

• SOAR Act applicants are less likely to have attended a low-performing school than DC students poten- 
tially eligible for the program, but as likely to have attended a charter school. In its “purpose” and “priority” 
sections, the SOAR Act (like its predecessor) emphasizes the goal of providing financial support to help disad- 
vantaged students leave low-performing DC public schools (specifically schools designated as “schools in need of 
improvement” or “SINI” under the federal Elementary and Secondary Education Act) to attend private schools. 
Across both school years (2011-12 and 2012-13) under the SOAR Act, the percentage of eligible applicants 
from SINI schools (64 percent) was lower than the percentage of income-eligible students in DC attending SINI 
schools (75 percent). A similar proportion of recent OSP applicants and school-age children estimated to be 
eligible for the program attended charter schools (34 percent and 35 percent, respectively) (table ES-1). 

• Most OSP applicants live in the lowest income neighborhoods in the District, where there are fewer partic- 
ipating private schools. Parents typically consider location an important factor when choosing a school for 
their child. 3 However, while 7 of 10 OSP applicants under the SOAR Act (69 percent total) live in wards with 
average annual household income below $100,000 (Wards 1, 5, 7, and 8), these same wards include less than 
half (43 percent) of participating schools. 


Table ES-1 Types of Schools Attended by Eligible Applicants at Time of Application Compared to Schools Attended by 
DC Students 



Eligible Applicants Under the SOAR Act 

DC Students in Public 
and Private Schools, 2010-11 

lype ot ocnooi Miienaea 

2011-12 

(%) 

2012-13 

(%) 

Both School 
Years (%) 

All, Regardless 
of Income (%) 

Potentially Eligible 
for OSP (%) 

All Public Schools 

74.1 

100.0 

82.2 

81.7 

96.1 

Traditional Public Schools 

44.6 

56.2 

48.3 

51.6 

61.0 

Charter Schools 

29.4 

43.8 

33.9 

30.1 

35.1 

Private Schools 

25.9 

0.0 

17.8 

18.3 

3.9 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 

SINI 

52.7 

89.6 

64.3 

61.3 

75.2 

Non-SINI and Private Schools 

47.2 

10.4 

35.7 

38.7 

24.8 

Total Percent 

100.0 

100.0 

100.0 

100.0 

100.0 

Total Number* 

906 

413 

1,319 

84,022 

53,072 


*Students entering kindergarten (N = 231) are excluded from the analysis because pre-Kindergarten students cannot be accurately designated as in public or 
private school programs. Totals do not sum to 100 percent due to rounding. Years refer to school years. 

SOURCES: Data for eligible applicants come from the Trust’s OSP Applicant database; data on all students come from the Common Core of Data (http:// 
nces.ed.gov/ccd/). SINI stands for schools identified as in need of improvement. Data for SINI designation for 2010-11 come from the DC Assessment and 
Accountability Data Report. SINI designations for 2011-12 and 2012-13 come from the Trust’s Lottery files. 


3 For example, Schneider and Buckley (2002) report that racial makeup, school location, test scores, and the basic programs offered, in this order, are factors 
in parent decisions. 
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Figure ES-5 Number of Eligible Applicants Awarded Scholarships, 2004-05 Through 2012-13 
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SOURCES: Data for 2004-05 through 2008-09 are from Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018); 
data for 2011-12 and 2012-13 are from the Trust’s OSP Applicant database. Years refer to school years. 


Scholarships: To what extent is the OSP enabling students to enroll in private schools? 

The purpose of the program is to help low-income DC residents attend private schools they might not otherwise 
be able to afford. But not all eligible students who apply to the program will receive an OSP scholarship. Nor will 
all students who receive a scholarship choose to use it. Identifying how many students are being given — and taking 
advantage of — the opportunity of a private school education through the program suggests three key findings: 

• The number of scholarships awarded varies over time. Since its launch in 2004, the OSP has awarded 5,288 
scholarships, with 1,330 awarded in the first two years of the SOAR Act. Annually, the number of awards has 
ranged from zero to 1,366. The number of scholarships awarded was less than the number of applicants when 
funding or open seats at private schools were limited and required the use of lotteries (2004—05, 2005—06, and 
2012-13) (figure ES-5). Between 2006-07 and 2008-09, funds were available only to fill in for students who 
had opted out of their scholarship. No awards were made during the 2 years when the program was closed to 
new applicants (2009-10 and 2010-11). 

• About 7 in 10 students who receive an OSP scholarship use it the following year to enroll in a private 
school. Under the SOAR Act so far, 72 percent of applicants receiving scholarships for the first time enroll in a 
private school, with an average across all years of 75 percent. The percentage of students who are offered scholar- 
ships but do not enroll (25-30 percent) is somewhat higher for the OSP under the SOAR Act than that of other 
voucher programs that previously operated in other cities. One factor that could contribute to this difference in 
use rates is that high school students are eligible for the OSP but were not in the other voucher programs, and 
lower proportions of these students use their vouchers compared to younger students (see next bullet item). 4 


4 See also table 3-7 in Wolf et al., Evaluation of the Opportunity Scholarship Program: Second Year Report on Participation, Institute of Education Sciences, 

2006. This table shows that first-year scholarship use rates for OSP students in grades 9-12 was 51 percent, compared to 69 percent for students in middle 
school (grades 6-8) and 79 percent for students in grades K-5 (elementary school). 
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• Older students, and those from disadvantaged schools and families, use the scholarship at lower rates than 
others. The SOAR Act identifies students applying from SINI schools as a priority group for scholarships. Yet 
these students were less likely to take up this opportunity when offered to them (65 percent of those awarded 
scholarships) than were eligible applicants from non-SINI public schools (73 percent) or from private schools 
(90 percent). Students who were awarded scholarships and whose parents were not employed full-time or were 
unmarried also used their scholarships at lower rates than students from more advantaged families. 
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1 Introduction 


The District of Columbia Opportunity Scholarship Program (OSP) — reauthorized under the Scholarships for 
Opportunity and Results (SOAR) Act of 2011 — continues to be a focal point in debates about school choice poli- 
cies. 5 Aimed at improving academic performance in the nations capital, the program provides tuition vouchers to 
low-income parents so that their children can attend private instead of public schools. While a few other cities and 
states also have voucher or private school scholarship programs, the OSP is the only one created and funded by the 
U.S. Congress. 

As part of the SOAR Act, Congress required an independent evaluation of the OSP, as it had done when the 
program was first created under the earlier DC School Choice Incentive Act of 2003. 6 This report is the first 
from the new evaluation and describes early implementation of the OSP under the SOAR Act. While the evalu- 
ations main goal is to assess the impact of the program on students’ academic progress, satisfaction, and safety, it 
will be a few years before those impacts can be determined. Until then, the evaluation will provide information 
describing the schools and families that are participating in the program, as important context for understanding 
those later results. This report explores three key questions about the schools, the applicants, and the award and 
use of OSP scholarships: 


Private Schools: How many participate , and what are their characteristics? 

School choice theory predicts that successful voucher programs will offer prospective parents an 
attractive set of school options and at the same time will need to offer private schools an attractive 
payment and set of procedures. Chapter 2 examines the private schools that participate in the OSP 
and how they compare with other private schools and public schools. 

Applicants: What is the nature of demand being generated for the program among 
eligible students and families? 

The 2011 re-authorization significantly increased funds for the program (from $14 million per year 
under the earlier statute to $20 million per year) in order to expand the availability of scholarships. 
Chapter 3 explores the number and types of OSP applicants across years and factors that may 
contribute to changes in demand. 

Scholarships: To what extent is the OSP enabling eligible students to enroll in 
private schools? 

The purpose of the program is to help low-income DC residents attend private schools they might 
not otherwise be able to afford. But not all eligible students who apply to the program will receive an 
OSP scholarship. Nor will all students who receive a scholarship choose to use it. Chapter 4 identifies 
how many students are being given — and taking advantage of — the opportunity of a private school 
education through the program. 


5 See http://www.gpo.gov/fdsys/pkg/BILLS-112hr471eh/pdf/BILLS-112hr471eh.pdf for the SOAR Act legislation. 

6 Final results from the earlier evaluation of the OSP can be found at Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa, and 
Matthew Carr. Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018). Washington, DC: National Center for Education Evalua- 
tion and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://ies.ed.gov/ncee/pubs/20104018/pdf/20104Q18.pdf . 
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Exhibit 1-1 Key Features of the OSP Defined in the SOAR Act 

Student eligibility criteria 

• DC resident 

• Income at or below 185 percent of the federal poverty line 

• Priority to students who: 

o Have a sibling already in program 
o Attended a low-performing SINI school 
o Offered a scholarship in the past but did not use it 
o Not already attending a private school 

Scholarship amount 

• $8,000 for grades K-8 

• $12,000 for grades 9-12 

Use of lotteries if there are more applicants than funds or space available. 


Overview of the OSP Under the SOAR Act 

The SOAR Act built on the earlier DC School Choice Incentive Act of 2003, which first established what is now 
called the OSP. As was the case in the previous statute, the SOAR Act laid out the purpose and operations of the 
program, including the criteria for student eligibility, the groups of students that receive priority for scholarships, 
the dollar amount of each scholarship, and the use of lotteries if the number of program applicants exceeded the 
number of available scholarships or open slots at participating schools (exhibit 1-1). 

However, the 2011 law included some changes that could affect the schools and students that participate in the 
OSP. These changes included: 

• Adding priority groups beyond just students in SINI schools, which could result in a different set of students 
being served and therefore to different overall outcomes. 

• Increasing the value of the voucher from $7,500 for all grade levels to either $8,000 (grades K-8) or 
$12,000 (grades 9-12), which might allow students to attend a broader range of private schools than in the 
past and families to spend less to supplement the voucher (where tuition exceeds the voucher amount). 

• Introducing more stringent accreditation and teacher quality requirements and fiscal controls that could 
change both the mix of DC private schools participating and the students educational experiences. 

In addition, the program has a new organization managing its operations. The original program manager closed its 
doors in 2010, and the U.S. Department of Educations Office of Innovation and Improvement (Oil) awarded a 
grant to the DC Children and Youth Investment Trust Corporation (“the Trust”) to take over the program. At the 
time data were collected and analyzed for this report, the Trust had completed 2 years of program implementation 
under the SOAR Act — Year 1 encompassed spring 2011 to spring 2012; Year 2, spring 2012 to spring 2013. 

The Trust recruited and determined the eligibility of new applicants, awarded scholarships to students, and helped 
families choose among the private schools. In addition, it notified schools of new requirements, issued voucher 
payments to schools, and continued to monitor the program. 
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Overview of the Evaluation 

The SOAR Act requires the U.S. Department of Educations Institute of Education Sciences (IES) to evaluate 
the school voucher program “using the strongest possible research design for determining the effectiveness of the 
opportunity scholarship program” (Sec. 3009 (2) (A) (i)). Westat and its partners, Pemberton Research and the 
University of California-San Diego, are carrying it out under contract to IES. 

The evaluation has several important features: 

• Use of lotteries to determine program impacts. The evaluation can take advantage of lotteries conducted 
when there is a surplus of applicants. With lotteries, only random chance — like the flip of a coin — determines 
which applicants receive a scholarship (the scholarship group) and which do not (the no-scholarship group). 
Researchers can track and compare outcomes for the two groups — e.g , their test scores, their satisfaction with 
and persistence in school — and be reasonably sure that differences between them are due to the program rather 
than to the characteristics or motivation of the students at the time they applied. This is especially important 
in the context of school choice because families wanting to apply for a choice program may have educational 
goals and aspirations that differ from the average family. 7 Using the lotteries makes the evaluation a “randomized 
control trial” (RCT) — considered the gold standard of scientific research. 

• Need for approximately 1,800 OSP applicants. To ensure that the random assignment does, in fact, create 
two groups that are similar prior to participation in the program, a large number of applicants need to enter the 
lotteries. Just as it is possible to flip a coin five times and get “heads” each time by chance (whereas flipping a fair 
coin a thousand times will result in nearly half of the flips being heads), comparing small samples of students 
can lead to unreliable conclusions. Statistical calculations suggest that the evaluation ideally be based on at least 
1,800 students who enter the OSP lotteries (see appendix A). 8 Because annual applications to the program have 
lagged behind what the Department expected, three lotteries of eligible applicants — in spring 2012, 2013, and 
2014 will be necessary for the evaluation to reach close to its targeted number. Data on those randomly assigned 
applicants will be collected through spring 2017. 

• Release of first impact report in 2016. The first year of program outcomes for the final group or “cohort” of 
applicants for the evaluation (those in the spring 2014 lotteries) cannot be measured until spring 2015. This 
means a first report describing early program impacts — the effects of participating in the OSP after 1 year) will 
be published a year later (spring/summer 2016). Under this plan, annual evaluation reports through 2015, 
starting with this one, will explore how the OSP is being implemented and other issues of interest. 9 


Overview of This Report 

The remainder of this report contains three chapters, addressing the three research questions identified earlier. 

In addition to describing the status of schools, applicants, and scholarships awarded, the report takes a look back 
to the beginning of the program in order to provide some perspective on how the program is now operating. 

To conduct this analysis, researchers drew on information from a variety of sources, ranging from the program 
operator s records and federal database of school characteristics to information on schools 5 own websites (see 
exhibit 1-2). Published reports from the prior evaluation of the OSP were also used, to allow for comparison to 
earlier years of the program. 


7 Randomized controlled trials have been conducted for a number of school voucher programs. Wolf et al. (2010) present findings for the DC OSP program. 

The Milwaukee voucher program was analyzed using lottery data by Rouse (1998) and Greene, Peterson, and Du (1999). Greene (2001) and Cowen (2008) 
analyzed the Charlotte scholarship plan. Howell, Peterson et al. (2002) and Howell, Wolff et al. (2002) studied the Dayton voucher program, a privately fund- 
ed voucher plan in Washington, DC, and a voucher program in New York. Finally, three studies used an RCT design to study a voucher plan in New York: Mayer 
et al., (2002), Barnard et al. (2003) and Krueger and Zhu, 2004). 

8 The prior evaluation of the OSP completed in 2010, was based on a sample of 2,308 eligible applicants who had entered lotteries and been ran- 
domly assigned. 

9 Appendix B provides more detail on the proposed schedule of evaluation data collection and reports, though the U.S. Department of Education is in the 
process of finalizing this plan. 
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Exhibit 1-2 Sources of Information for the Report 



NOTE: SINI stands for school in need of improvement. Years refer to school years. 


4 




EVALUATION OF THE DC OPPORTUNITY SCHOLARSHIP PROGRAM 

An Early Look at Applicants and Participating Schools Under the SOAR Act 


The Trust s program staff collects application forms from interested families and issues voucher payments to 
schools for each scholarship recipient confirmed as attending; there is no other source of data on applicants or 
scholarship usage and amounts. In addition, the Trust maintains a file of participating OSP schools dating back to 
2004 (when the first OSP began) and obtains information annually from participating private schools to provide a 
“school directory” to parents. 10 

The evaluation team supplemented Trust records with other information. IES 5 National Center for Education 
Statistics (NCES) compiles a census of basic information about every public school in the United States from state 
education agencies (the CCD) annually, and about every private school in the country (the PSS) from the indi- 
vidual schools biennially. The report uses these data to examine characteristics of public and private schools in DC 
attended by program applicants and by DC students overall. The report also uses data from NCES’ 2011-12 SASS 
and the 2010 census, including 3-year estimates provided by the American Community Survey (ACS), to explore 
the percentage of students likely to be eligible for the program that actually apply. 

Finally, the evaluation team added to information from the Trust s school directory by extracting data directly from 
DC private schools’ websites. This approach was used to obtain school location, confirmation that a participating 
school is still operating, and religious affiliation of non-participating schools. * 11 

Throughout the report, tables and figures compare schools and students using simple percentages and averages. 
Because our analysis is based on all DC private and traditional public and charter schools and all OSP appli- 
cants, rather than a subset of them, the “statistical significance 55 of differences between groups was not calculated 
(i.e., there is no sampling error to take into account). 

Although the report uses the best available data to describe OSP schools, applicants, and scholarships awarded 
under the SOAR Act, there are some limitations. First, data were obtained from different sources and some- 
times different time periods; for example, the years represented by the most current NCES public and private 
school databases are not exactly aligned (2010—1 1 vs. 2009—10). Second, we frequently compare the status of 
the OSP in the year or two since the SOAR Act was passed to the status in a particular year under the earlier 
statute (2005-06), for context. We chose the comparison year because: (1) both time periods were early after their 
respective statutes were passed and when significant funds were available for new scholarships, and (2) compre- 
hensive data from the earlier evaluation were available for the 2005—06 year that was not available for other years. 
However, 2005—06 is not necessarily representative of every other year under the earlier statute. 


10 Data provided by the Trust were not independently verified. 

11 Data on published tuition rates for non-participating private schools could not be reliably obtained from school websites. 
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2 Private Schools 

How Many Participate and What Are Their Characteristics? 


The first 2 years of the reauthorized program provide a useful opportunity to examine the number and mix of 
schools that participate in the OSP and how they might be changing, in response to revisions in the 2011 law or 
for other reasons. For example, more stringent accountability requirements might discourage private schools from 
participating, but higher voucher amounts might encourage more participation. Changes in school participation 
could have implications for program participants; schools’ leaving the program could force transfers and disrupt 
student learning. Overall, the number and variety of private school options may affect parent choices, with cost 
and location (nearness to home) important factors in those decisions. 12 

The analysis leads to four key findings about private school involvement: 

Just over half of all DC private schools participate, with a smaller core set participating 
since the program began. 

The number of schools participating in the OSP has fluctuated. Some of the shifts reflect changes in the private 
school market, but factors related to the program may also be at work. 

• Fifty-two private schools in DC currently participate in the program. This number has shifted over time, ranging 
from 57 in the first year the program was operating, to a high of 68 the second year, and to 52 in the most 
recent year under the SOAR Act (figure 2-1). The current schools enroll about 12 percent of all private school 
students in DC. 13 

• The decline in the number of participating schools goes beyond the contraction in the overall private school 
market in DC. The total number of private schools in DC dropped by 13 percent from a high of 109 in 
2004-05 to 95 in 2012-13 (figure 2-1), perhaps reflecting the same factors that have led to a reduction in 
private school enrollments nationwide (Ewert 2013). However, even as a share of the private schools oper- 
ating in DC each year, participation in the OSP has fallen off: after the first year when 57 of 1 09 DC private 
schools (52 percent) signed on to the OSP, the share of private schools involved in the OSP has declined from 
65 percent (2005-06) to 55 percent (2012-13) (figure 2-2). These trends suggest that the changes to the 
program made by Congress in 201 1 have not drawn more private schools to participate. 14 

• Various circumstances appear to have led schools to stop participating in the OSP. The largest number of schools 
exited the program in 2007-08 and in 2008-09. In 2007-08, seven participating Catholic schools converted to 
public charter schools and could no longer receive vouchers through the program. The next year, two additional 
participating schools converted to charter schools; four were no longer operating; and five withdrew from the 
program. Some schools may have stopped operating because of financial instability and declining enrollment 
(Labbe 2007, 2008). Uncertainty over continued program funding also may have contributed to some schools’ 
decision to leave the program as Congress publicly debated and then stopped funding new OSP scholarships in 
December 2009. 15 


12. See, for example, Schneider and Buckley (2002). 

13. Excluding pre-K and Kindergarten, total private school enrollment in DC is 13,194, while enrollment in the 52 OSP schools is 12,401 students, according to 
the most recent data available (Private School Universe Survey (2009-10); see http://nces.ed.gov/surveys/pss/pssdata.asp ). 

14. The data do not support drawing conclusions about whether participating in the program prevents a private school from having to close. 

15. See, for example, http://www.washingtonpost.com/wp-dyn/content/article/2007/ll/05/AR200711050Q433.html 
http://articles.washingtonpost.com/2008-04-ll/news/36827864_l_catholic-schools-charter-schools-conversion-schools 
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Figure 2-1 Total Number of DC Private Schools and Private Schools Participating in the OSF? 
2004-05 through 2012-13 



04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12 12-13 


Years Schools Participated 


Total Non-Participating 
Private Schools 

Total Participating 
Private Schools 


*2009-10 and 2010-11 were after the prior evaluation stopped collecting data. There are no reliable historic records of how many private 
schools existed during those years. Years refer to school years. 

NOTE: The total number of private schools operating in DC each year is represented by the number at the top of each bar, with the bar divided 
into the number not participating in the OPS (top, blue shading) and the number participating (bottom, grey shading). So, for example, in 
2004-05, out of 109 private schools in DC, 52 did not participate in the OSf? while 57 participated. 

SOURCES: Data for the total number of private schools are from Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 
2010-4018), the PSS (http://nces.ed.gov/surveys/pss/pssdata.asp ) and private school websites. Data for participating schools are from the 
Trust’s Participating School File. 


Figure 2-2 Number of Private Schools Leaving, Joining, and Continuing in the OSF? 
2004-05 Through 2012-13 



NOTE: Years refer to school years. 

SOURCE: Data for 2004-05 through 2012-13 are from the Trust’s Participating School File. 
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Figure 2-3 Percentage of Participating Private Schools That Enrolled OSP Students, 
2004-05 Through 2012-13 
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SOURCES: Data for 2004-05 through 2008-09 are from Evaluation of the DC Opportunity Scholarship Program: Final Report 
NCEE 2010-4018); data for 2011-12 and 2012-13 are from the Trust’s OSP Payment files. Years refer to school years. 


• Although at least 50 private schools have participated each year the OSP has been operating, a core set of 
33 private schools has participated every year. 

• More than 80 percent of participating schools enroll at least some OSP students, but the most recent school year 
(2012-13) was an exception (figure 2-3). Private schools “participate” by signing on to accept OSP vouchers as 
tuition payment, but will not enroll an OSP student if none chooses to apply or if none of those applying meets 
the schools admission requirements. 16 It is unclear why fewer participating schools are enrolling OSP students 
under the new program created by the SOAR Act. 


Participating schools are more likely now than in the past to report tuition rates above the 
OSP scholarship amounts, to have no religious affiliation, to serve grades 9-12, and to have 
less diverse student populations. 

The characteristics of the overall pool of schools now available to OSP families differ from the earlier years of the 
program, when comparing years for which the same data are available (figure 2-4). These differences may reflect 
both a distinction in the set of schools that participated earlier versus more recently and changes in the schools 
participating on an ongoing basis. 17 

• A larger proportion of participating schools under the SOAR Act have published tuition rates above the 
legislated scholarship maximums than did schools participating under the earlier DC Choice Incentive Act 


16. By law private schools may apply their standard admissions procedures to OSP students seeking to enroll, except the schools cannot discriminate on the 
basis of race, color, national origin, religion, sex, or disability status (clarifications added in the SOAR Act). 

17. The characteristics of participating schools presented here treat each school equally. When schools are weighted by the number of OSP students they serve, 
the gaps between schools participating in 2005-06 and 2011-12 are smaller but still mostly persist; the proportion of schools that are religiously affiliated 
is no longer different across the two time periods. The smaller gaps or the elimination of them is partly due to the number of higher tuition, independent 
private schools that have signed up to participate in the OSP since the early years of the program but that do not serve OSP students or serve a very small 
number of them. See appendix C for more detail on how the characteristics of participating schools has changed over time when the number of students 
served is taken into account. 
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Figure 2-4 Characteristics of Particpating Private Schools Under the DC School Choice Incentive Act and the 
SOAR Act, 2005-06 and 2011-12 


100 



Percent Serving Percent Reporting Average Percent Percent Religiously Percent Catholic 
Grades 9-12 Tuition Rates Minority Students Affiliated 


Above 0SP Cap Served* 

School Characteristic 


Incentive Act 2005-06 


SOAR Act 2011-12 


*“Average Percent Minority Students Served” is calculated as the mean of the percent of each school’s student body that is from a minority 
racial-ethnic group (non-White). 

NOTE: The 2011-12 school year (under the SOAR Act) and the 2005-06 school year (under the DC School Choice Incentive Act) are used to 
compare the characteristics of the program under the two statutes, with these specific years chosen because: (1) both time periods were soon 
after each statute was passed and when significant funds were available for new scholarships, and (2) comprehensive data from the earlier 
evaluation were available for the 2005-06 year that were not available for other years. Tuition data for participating schools in 2005-06 are 
available for 60 of the 68 schools and for all 53 participating schools in 2011-12. Years refer to school years. 

SOURCES: Data for 2005-06 come from Evaluation of the DC Opportunity Scholarship Program: Second Year Report on Participation (NCEE 
2006-4003); 2011-12 data are from the Trust’s OSP Participating School Directory and private school websites. 


(64 percent in 201 1-12 versus 39 percent in 2005-26). 18 Currently, similar proportions of schools serving 
students in grades 9-12 (68 percent) and in grades K-8 (62 percent) publish tuition rates above the OSP cap 
(figure 2-5). Future reports will examine whether OSP parents are charged the published tuition rate and, if so, 
how parents pay the difference, taking into account patterns suggesting that OSP students cluster in partici- 
pating schools that do not charge tuition above the cap. 19 

• On average, schools most recently participating are serving a smaller proportion of students from racial-ethnic 
minority groups (65 percent in 2011-12 versus 77 percent in 2005-06). 

• The percentage of participating schools serving high school students has increased under the SOAR Act 
(36 percent) compared to an earlier year of the program — 2005-06 (22 percent). However, there is little 
evidence that the higher value of the voucher in the new law is a driving factor: of the four schools that joined 
the program during this interval, three of them did so before the passage of the SOAR Act when the scholarship 
amount for high school students rose from $7,500 to $12,000. 20 


18. Analysis of tuition data for 24 schools that participated in both 2005-06 (under the DC School Choice Incentive Act) and in 2011-12 (under the OSP) 
indicates their average published tuition rose from $10,329 to $15,162. This nearly 50 percent overall increase was equivalent to a 6.6 percent increase 
per year — higher than the 5.3 percent per year private school tuition increase for the country as a whole between 2003 and 2007 (the most recent years 
for which these data have been collected, see Digest of Education Statistics, table 71). By contrast, the Consumer Price Index rose 2.3 percent a year from 
2005 to 2011. 

19. See, for example, figure C-2 in appendix C. 

20. Among the four schools serving students in grades 9-12 that joined since the OSP started, two serve students in grades 6-12. Of the two schools serving 
only students in grades 9-12, one joined the program since passage of the SOAR Act. 
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Figure 2-5 Percentage of Private Schools With Published Tuition Above and Below the Voucher Amount, 
for Schools Participating Under the SOAR Act, 2011-12 
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(n = 34) (n = 19) (n = 53) 

Grade Levels Served 


Above the 
voucher amount 
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NOTE: Schools were included in the category corresponding to the highest grade served. Of the 19 schools serving students in grades 9 -12, 13 
also served K-8 students. Years refer to school years. When the proportion of schools is weighted by the number of OSP students served, the 
share charging above the voucher amount is much smaller but similar for both K-8 and 9-12 schools (see figure C-2 in appendix C). 

SOURCE: Data are from the Trust’s 2011-12 OSP Participating School Directory. 


• Proportionally fewer schools participating under the SOAR Act in 201 1-12 than in 2005-06 are affiliated 
with a religious denomination (62 percent versus 68 percent previously). During this period, there was a 
decline in the number of both Catholic schools (28 to 20) and other faith-based schools (18 to 13) participating 


in the OSP. 


Private schools that currently participate in the OSP have been operating longer, are more 
likely to be religiously affiliated, and have larger class sizes than other private schools in DC. 

The DC private school sector is diverse, with schools that vary in their selectivity, target populations, affiliation, and 
other characteristics. 21 Available data suggest that OSP schools are a distinct group of DC private schools (table 2-1). 

• A higher share of participating schools than non-participating private schools is religiously affiliated (64 percent 
versus 29 percent). 

• Compared to private schools that do not accept OSP vouchers, participating schools have been operating 
longer (on average 76 years versus 50 years), are larger (average enrollment of 243 versus 104), and have higher 
pupil- teacher ratios (on average nine students per teacher versus almost seven). These differences largely reflect 
the higher concentration of Catholic schools participating in the OSP (38 percent compared to 7 percent not 
participating), since schools under the Archdiocese are older and have larger teacher-pupil ratios and overall 
enrollments than do other types of private schools. 22 


21. DC private schools include highly selective institutions, schools targeted to students with significant disabilities, schools training students for the performing 
or fine arts, those associated with the Catholic Archdioceses and other faiths, and those that are non faith based and independent. 

22. Participating Catholic schools are older (104 years on average versus 60 years for non-faith-based and 54 years for other faith-based schools) and have 
higher pupil/teacher ratios (about 11 students per teacher on average) compared to other faith-based schools and non-faith-based schools, which have 9 and 
7 students per teacher on average. Catholic schools enroll about the same number of students as other faith-based schools (300 versus 297 students on 
average), almost twice as many as enrolled by non-faith-based schools (150 students on average). 
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Table 2-1 Characteristics of Participating and Nonparticipating DC Private Schools and Traditional Public 
and Charter Schools, 2011-12 



DC Private Schools 

Participating Non-Participating 

DC Traditional Public 
and Charter Schools 

Percent serving high school 

35.8 

33.3 

23.8 

Average pupil-teacher ratio 

9.0 

6.6 

12.4 

Average years in operation 

75.8 

49.8 

N/A 

Average student population 

243 

104 

348 

Percent minority students 

65.2 

58.6 

93.6 

Percent faith based 

64. 

37.7 

28.5* 

Percent Catholic 

7.1* 

N/A 

N/A 

Percent non faith based 

35.8 

71.4* 

100 

Number of schools overall 

53 

30 

206 


*Data were available for 28 non-participating schools. N/A indicates data were not available. Years refer to school years. 

SOURCES: Private school data are from the Trust’s OSP Participating School Directory, private school websites, and the Private School Universe 

Survey http://nces.ed.gov/surveys/pss/pssdata.asp ; data for traditional public and charter schools are from Common Core of Data ( http://nces.ed.gov/ccd/ ). 

Compared to the public schools parents may be considering, participating schools are smaller, 
serve a higher share of White students, and are clustered in affluent areas of the city. 

Parents of eligible applicants, if they receive a scholarship, have choices from among participating private schools 
and a wide variety of public schools, including charter and magnet schools spread across the district. These parents 
may view the characteristics of the private schools described above, available from public sources, as “proxies” for 
school quality because test scores cannot be compared across sectors. 23 OSP schools differ in several ways from 
traditional public schools and charter schools (table 2-1). 

• Participating private schools are smaller (average enrollment of 243 versus 348) and have lower pupil-teacher 
ratios than traditional public schools and charter schools (9 students per teacher versus 12 students per teacher). 
Participating schools also have a higher proportion of White students than public schools (35 percent versus 

6 percent). 

• Parents often consider location an important factor when choosing a school for their child. 24 The District of 
Columbia is divided into eight wards that represent geographic clusters of neighborhoods. 23 Participating private 
schools are less widely dispersed compared with public schools and may require more travel time for students 
and parents. Thirty of 53 OSP schools (57 percent) in 201 1-12 were located in the four most affluent sections 
of the city (in wards with average annual household income above $100,000). 26 The remaining 23 schools 

(43 percent) were evenly distributed across the city’s other four wards, where household income ranged from 
$44,000 to $94,000 (figure 2-6). 27 The match between where participating schools are located and where appli- 
cants reside is explored in chapter 3 of the report. 


23. For example, Schneider and Buckley (2002) report that racial makeup, school location, test scores, and the basic programs offered, in this order, are factors 
in parent decisions. 

24 In future reports, the evaluation will explore which factors OSP parents indicate they value in choosing schools. 

25 See http://www.neighborhoodinfodc.org/index.html for profiles of each ward in D.C. 

26 Average annual household income among families living in Wards 2, 3, 4, and 6 is $205,000, $257,000, $117,000 and $116,000, respectively. 

27 Ward 3 includes a large percentage of the city’s wealthiest neighborhoods, including American University Park, Chevy Chase, Forest Hills, Friendship Heights, 
Palisades, and Spring Valley. Ward 4, the northernmost section of the city, includes Barnaby Woods, portions of Chevy Chase, Shepherd Park, and Takoma. 
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Figure 2-6 Location and Size of Private Schools Participating in the SOAR Act OSF? by Ward, 2011-12 


SOURCE: Trust’s OSP Participating School Directory. 
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3 Applicants 

What Is the Nature of Demand for the Program 
Among Eligible Students and Families? 

Ongoing interest in the OSP among families is clearly critical to the programs success. The main available indi- 
cator of family interest is applications for scholarships. But because the number and type of eligible applicants also 
reflect the decisions and efforts of the program operator and of the U.S. Department of Education, applications 
are best considered measures of “generated” demand, while the true demand for the program is unknown. 28 Other 
influences on OSP demand may include the debates about the program and its sustainability in the media, partic- 
ipating students’ own experiences in the program — positive or negative — through word of mouth or if students 
give up their scholarships and make space available for new students, 29 or the increasing availability of other choice 
options in DC (e.g., charter schools). 

Analyzing how many and which students and families apply to participate in the OSP can still provide valuable 
information about demand, key factors that may be increasing or limiting that demand, and whether the program 
is reaching its primary target group — students who were attending low-performing SINI public schools. There are 
several key findings about the number and characteristics of eligible applicants recruited: 

The number of applications has fluctuated, mostly along with funds available to admit new students. 

Since the OSP began, the number of applications from eligible students has varied (figure 3-1). The number 
appears to be linked to key milestones in program operations. 

• More students and families applied in the years immediately after the two congressional authorizations in 2003 
and 2011 than in other years. The highest numbers of applications were in 2004-05 (1,848 applications) and 
2005-06 (2,199 applications) — when the program first opened — and 2011-12 (1,014 applications) when the 
program re-opened to new students. In these periods, the program operator had fewer continuing students to 
support and is likely to have recruited aggressively due to the large amount of funding available for new scholar- 
ships. Alternatively, these may be periods when “pent-up” demand (desire to participate among families who had 
not previously applied) could have been strongest. 

• In contrast, in other years (2006-07 through 2008-09, the program operator scaled back recruitment because 
OSP funds were available only to replace students leaving the program. 30 Most of the available OSP resources 
were used to support tuition for students already in the program and attending a participating school. There 

is some evidence that in the 2006—07 through 2008—09 rounds, the program operator recruited just enough 
eligible applicants to take up the remaining available funding. 31 For 2009-10 and 2010-1 1, no applications 
were taken because Congress had closed the program to new students and provided funds only to support 
continuing scholarship recipients. 

• Despite the greater funding available under the SOAR Act (close to $20 million per year), there have been fewer 
eligible applicants than during the comparable period under the earlier statute (when close to $14 million per 
year). This suggests that program funding is not the sole determinant of demand, and other factors, such as those 
described above, probably play a role. 

28 For the purpose of this report’s analysis, for example, eligible applicants are only those students eligible to receive a scholarship in a given year. For example, 
in some years, otherwise eligible students who were already attending private schools when they applied to the OSP were not permitted to enter lotteries to 
determine the award of scholarships because the legislation allows for targeting of OSP funds to families that cannot exercise school choice. 

29 See, for example, http://www.washingtonpost.com/blogs/dc-wire/post/boehner-lieberman-ask-obama-to-fund-dc-scholarship-program/2012/03/22/ 
glQAPPICUS_blog.html ; http://www.washingtonpost.com/blogs/dc-wire/post/obama-opposes-boehners-dc-school-choice-bill-but-does-not-threaten-ve- 
to/2011/03/29/ AFj0O0wB_blog.html ; http://www.washingtonpost.com/politics/house-speaker-boehner-promotes-school-funding-program-in%20dis- 
trict/2011/02/22/ ABzubTX_story.html 

30 See p. 26 of a report prepared by the Government Accountability Office on the OSP under the DC School Choice Incentive Act, available at http://www.gao. 
gov/products/GAQ-08-9 . 

31 During those years, the number of eligible applicants reported by the program operator was exactly equal to the number of awards made. See Evaluation of 
the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018; table 1.1, p. 3). 
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Figure 3-1 Number of Eligible Applicants, 2004-05 Through 2012-13 
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SOURCES: Data for 2004-05 through 2008-09 are from Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018); 
data for 2011-12 and 2012-13 are from the Trust’s OSP Applicant database. Years refer to school years. 


Applicants under the SOAR Act represent fewer than 5 percent of eligible DC students. 

The law sets clear residency and family-income criteria to determine student eligibility. Students must be residents 
of DC and have an annual household income within 185 percent of the 2011 federal poverty level ($34,281 for 
a family of three and $41,348 for a family of four). From available data, it is impossible to determine whether 
eligible families and students that do not apply do not know about the OSP, are not interested in the OSP 
(< e.g ., are satisfied with their current school arrangements), or are not able to complete the steps required to apply 
to the program. 

• There are almost 80,000 children who reside in the District of Columbia in the age ranges served by the OSP. 32 
However, the number of these children who are in households with income at or below 185 percent of the 
federal poverty level is not known precisely. One possible proxy is the number of children eligible for the federal 
free or reduced-price lunch, which uses the same income eligibility criterion as the OSP. That number is esti- 
mated to be about 53,000 children in DC. 33 

• The 1,550 OSP eligible applicants under the SOAR Act represent between 3 and 4 percent of those 53,000 
students estimated to be eligible for the program. 


32 The American Community Survey reports the number of school-age children (age 5a-17) in DC in 2012 to be 11.2 percent of the total population, which yields 
70,820 school-age children. Some preschool children who are eligible to apply to the OSP to attend kindergarten can be 4 years old. Including all 4-year olds, 
we estimate that 78,660 children fall within the 4-17 age range. 

33 In the NCES Common Core Data Survey (2010) DC public schools reported 50,982 students eligible for the federal free or reduced-price lunch program. How- 
ever, counts of students eligible for the federal lunch program are not available from the NCES Private School Survey (PSS). PSS reported a total of 15,368 
private school students in DC. We applied to this total the national proportion of students in urban private schools that are eligible for the federal lunch 
program (13.6 percent) from the most recent NCES SASS (see http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2013312 , p. 8). We used the national 
average because the SASS, both the public school and private school versions, is based on national samples of schools and does not provide reliable data 
specific to DC schools. The resulting estimate is that 2,090 private school students in DC are eligible for free or reduced-price lunches. When added to the 
reported number of eligible students in DC public schools, the total is 53,072 students. This estimate may overstate the number of children in households 
with incomes that qualify them for both the OSP and the federal lunch program, for a number of reasons. First, schools may over- certify children as eligible 
(as reported by a 2007 study for the U.S. Department of Agriculture, Food and Nutrition Service, which runs the federal lunch program). Some schools use 
‘Provision 2’ or ‘Provision 3’ of the lunch program rules and certify all their students to be eligible for free- or reduced-price lunches if most of them are, 
which may lead to some individual children being certified as eligible even though their household income exceeds the poverty threshold. In addition, the 
number of private school students in DC in the PSS probably includes some students who are not DC residents and would therefore be ineligible for the OSP 
Our estimate may also understate the total number of children who might be eligible for the OSP if DC private schools do not fully certify eligible students for 
the federal lunch program. 
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So far, SOAR Act applicants are less likely to have attended a low-performing school than DC 
students potentially eligible for the program, but as likely to have attended a charter school. 

In its “purpose” and “priority” sections, the SOAR Act (like its predecessor) emphasizes the goal of providing 
financial support to help disadvantaged students leave low-performing DC public schools (specifically schools 
designated as SINI under the federal Elementary and Secondary Education Act) and attend private schools. The 
statute s overarching aim is to provide expanded school options for families which, in the District of Columbia, 
means increasing the choices already available through the charter school sector. Examining the types of schools 
attended by applicants, DC students, and eligible DC students provides information about the extent of progress 
toward these objectives. 

• Nearly two-thirds of eligible OSP applicants in the first 2 years under the SOAR Act (64 percent) attended SINI 
schools at the time of their application to the program (table 3-1). The percentage designated as SINI was much 
higher in the second year of the program (almost 90 percent). Students attending private schools comprised a 
quarter of eligible applicants for scholarships in the first year but none in the second year. 34 

• Across both school years (2011-12 and 2012-13) under the SOAR Act, the percent of eligible applicants from 
SINI schools (64 percent) was lower than the percent of income-eligible students in DC attending SINI schools 
(75 percent), though it was similar to the share of all students in DC who attended SINI schools (61 percent). 

If the eligible applicant pool in future years is more similar to the second rather than first year under the 
SOAR Act, then the program would be serving a higher fraction of low-income students from SINI schools 
(90 percent) than is typical in the city (75 percent). 

• A similar proportion of recent OSP applicants and DC school-age children estimated to be eligible for the 
program attended charter schools (34 percent and 35 percent, respectively). It is important to note that the 
number of students estimated to be eligible includes low-income students attending private schools, making 
the share of DC students attending charter schools in table 3-1 lower than the published rates of charter school 
attendance among DC public school students only. 

Compared to those in earlier years, recent applicants under the SOAR Act are better educated 
and employed, but less likely to be married. 

Research suggests that family configuration and resources other than income can influence student outcomes that 
could have implications for the effectiveness of the program. 35 Comparing the characteristics of OSP applicants 
under the SOAR Act (201 1-12 and 2012-13) with those of applicants who applied to the program for 2005-06 
provides a complicated picture of whether recent applicants are better off or not. 

• Parents of applicants in 201 1-12 and 2012-13 were more educated than parents of applicants in the earlier 
years of the program (figure 3-2). For example, 60 percent of later applicants had parents with at least some 
college experience compared with 46 percent of parents of earlier applicants. Parents of recent SOAR Act appli- 
cants had an average of 14 years of schooling compared with 13 years for earlier applicants. 

• Parents of newer applicants are more likely to be employed full-time (26 percent) than parents of applicants in 
earlier years of the program (1 1 percent). 

• On the other hand, a smaller share of recent applicants (22 percent) than earlier applicants (63 percent) have 
parents who are married. 


34 In 2012-13, 95 students already attending private schools were deemed eligible based on their residency and family income, but these students were in 
the lowest priority group defined by the SOAR Act and given a 0 percent probability of receiving a scholarship in the program lottery. While they were eligible 
for the program, they were not eligible for scholarships that year and therefore were not included in the count of eligible applicants. In contrast, in 2011-12, 
there were no program lotteries, and all students deemed eligible for the program were also eligible for scholarships that year. 

35 For example, Peng and Wright (1994) report that home environmental and educational activities contributed significantly to the differences in achievement, 
and Nord (1998) found that students in two-parent families are more likely to get As when their fathers are involved in school. 
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Table 3-1 Types of Schools Attended by Eligible Applicants at Time of Application Compared to Schools Attended 
by DC Students 


TimA Af C^UaaI A a rl a ri 

Eligible Applicants Under the SOAR Act 

DC Students in Public 
and Private Schools, 2010-11 

iype or ocnooi Aiienueu 

2011-12 

(%) 

2012-13 

(%) 

Both School 
Years (%) 

All, Regardless 
of Income (%) 

Potentially Eligible 
for OSP (%) 

All Public Schools 

74.1 

100.0 

82.2 

81.7 

96.1 

Traditional Public Schools 

44.6 

56.2 

48.3 

51.6 

61.0 

Charter Schools 

29.4 

43.8 

33.9 

30.1 

35.1 

Private Schools 

25.9 

0.0 

17.8 

18.3 

3.9 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 

SINI 

52.7 

89.6 

64.3 

61.3 

75.2 

Non-SINI and Private Schools 

47.2 

10.4 

35.7 

38.7 

24.8 

Total Percent 

100.0 

100.0 

100.0 

100.0 

100.0 

Total Number* 

906 

413 

1,319 

84,022 

53,072 


*Students entering kindergarten (N = 231) are excluded from the analysis because pre-Kindergarten students cannot be accurately designated as in public or 
private school programs. Totals do not sum to 100 percent due to rounding. Years refer to school years. 

SOURCES: Data for eligible applicants come from the Trust’s OSP Applicant database; data on all students come from the Common Core of Data 
(http://nces.ed.gov/ccd/) . SINI stands for schools identified as in need of improvement. Data for SINI designation for 2010-11 come from the 
DC Assessment and Accountability Data Report. SINI designations for 2011-12 and 2012-13 come from the Trust’s Lottery files. 


Figure 3-2 Parent Characteristics of OSP Applicants Under the Incentive Act and the SOAR Act, 
2005-06, and 2011-12 and 2012-13, Combined 



Average Years Attended Any Employed Married 

of Schooling College Full-time 


Parent Characteristics 


Incentive Act 2005-06 


SOAR Act 2011-12 
and 2012-13 


SOURCES: Data for 2005-06 are from the Restricted Use File prepared for the Evaluation of the DC Opportunity Scholarship Program , 2010 ; data 
for 2011-12 and 2012-13 are from the Trust’s OSP Applicant database. Years refer to school years. 
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Most OSP applicants live in the lowest income neighborhoods in the District where there are 
fewer participating private schools. 

Comparing the distribution of OSP schools by ward, shown earlier in figure 2-6, with the distribution of OSP 
applicants’ residences by ward provides a measure of the convenience of the schools. For recent OSP applicants, 
the match between where they live and where the OSP schools are located suggests some transportation effort is 
needed (figure 3-3). 

• Over two-thirds of OSP applicants (69 percent total) live in wards with average annual household income 
below $100,000. Over 40 percent of applicants live southeast of the Anacostia River in Wards 7 and 8 (21 and 
21 percent), which have the lowest average annual household income of all of the wards in DC ($55,000 and 
$44,000, respectively). The remaining 27 percent of applicants live in Wards 1 and 5, which have average annual 
household income of $94,000 and $79,000, respectively. 

• Although 69 percent of the applicants live in Wards 1, 5, 7, and 8, just over 40 percent (43 percent) of partici- 
pating schools are located in these neighborhoods. 
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Figure 3-3 Location of Participating Private Schools and Eligible Applicants’ Residence, by Ward, 
2011-12 and 2012-13, Combined 
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SOURCES: School data are from the Trust’s OSP Participating School Directory; data about eligible applicants are from the Trust’s OSP Applicant 
database. 
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4 Scholarships 

To What Extent Is the OSP Enabling Students 
to Enroll in Private Schools? 

The OSP model uses a three-step process by which the goal of private-school attendance can become reality. First, 
families must apply and be found eligible. Second, the child must be awarded a scholarship, which may or may 
not happen, depending on the number of applications in a given year and whether a lottery is used to determine 
who is awarded a scholarship. Third, the family whose child receives a scholarship must decide to enroll him or her 
in a participating school. 

The extent to which scholarships are awarded and used — the last two steps in the process — is important to both 
the families who apply and the schools that select and receive OSP scholarship recipients. This information 
can also contribute to program improvement because low use of scholarships awarded could signal a mismatch 
between student needs and what the program offers. 

Among the key findings: 

The number of scholarships awarded varies over time. 

Between the OSP’s launch in 2004 and 2012-13, the program has awarded 5,288 scholarships, with 1,330 
awarded in the first 2 years of the SOAR Act. Annually, the number of awards has ranged from zero to 1,366. 36 
Because the number of new scholarships awarded in any year cannot exceed the number of eligible applications 
received or the funds available to support new program participants, the two measures — eligible applications and 
awards — often, but not always, go up and down together. 

• The number of scholarships awarded was less than the number of applications received from eligible students 
when lotteries were needed to allocate OSP vouchers (2004-05, 2005-06, and 2012-13) (figure 4-1). 

• Between 2006—07 and 2008—09, award numbers were lower and matched the number of eligible applications. 

In those years, funds were available only to fill in for students who had opted out of their scholarships. 

• No awards were made during the 2 years when the program was closed to new applicants (2009-10 and 
2010 - 11 ). 

About 7 in 10 students who receive an OSP scholarship use it the following year to enroll in a 
private school. 

There are many reasons why students awarded an OSP scholarship do not use it. These issues were explored in 
the earlier evaluation of the program and will continue to be a focus as the current evaluation goes forward. It is 
important to note that use rates are not indicators of the effects of being awarded a scholarship on private school 
enrollment because some applicants might find a way to enroll even without one, another issue that will be exam- 
ined in later reports. 

• Under the SOAR Act, 72 percent of applicants receiving scholarships enroll in a private school the following 
year, compared with an average across years of 75 percent (figure 4-2). 

• The percentage of students who are offered scholarships but do not enroll (25—30 percent) is somewhat higher 
for the OSP under the SOAR Act than that of other voucher programs that previously operated in cities such as 
New York City (18 percent) and Dayton, Ohio (22 percent), and slightly lower than that for a privately funded 


36 The total number of scholarships awarded under the Incentive Act OSP is 3,738. The total numbers of scholarships awarded in the first 2 years under the 
SOAR Act OSP is 1,550. 
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Figure 4-1 Number of Eligible Applicants Awarded Scholarships, 2004-05 Through 2012-13 
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SOURCES: Data for 2004-05 through 2008-09 are from Evaluation of the DC Opportunity Scholarship Program: Final Report (NCEE 2010-4018); 
data for 2011-12 and 2012-13 are from the Trust’s OSP Applicant database. Years refer to school years. 


Figure 4-2 Percentage of Offered Scholarships Used in Initial Year, 2004-05 Through 2012-13 



SOURCES: Data for 2004-09 are author’s calculations based on information from Evaluation of the DC Opportunity Scholarship Program: Final 
Report (NCEE 2010-4018); data for 2011-13 are from the Trust’s OSP Payment files. Years refer to school years. 


choice program operated in DC in the late 1990s (32 percent). One factor that could contribute to the differ- 
ence in use rates between DC and other cities — especially New York — is that middle and high school students 
are eligible for the OSP but were not in the New York program, while the privately funded DC program was 
available to children in K-8. Lower proportions of older students use their vouchers compared to younger 
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Figure 4-3 Percentage of Applicants Offered Scholarships Who Use Them, by Type of School at the 
Time of Application, 2011-12 and 2012-13 Combined 



NOTE: “SINI” and “Non-SINI” designations refer to traditional public and charter schools, N = 779. For private schools, N = 235. 
SOURCES: School-related data are from the Trust’s OSP application database and lottery files, and the Common Core of Data (http://nces. 
ed.gov/ccd/) ; data about scholarship use are from the Trust’s OSP Payment files. Years refer to school years. 


students (see figure 4 -4). 37 The use of OSP scholarships is somewhat lower under the SOAR Act than during 
the comparable period under the earlier Incentive Act (about 74 percent, the average of 75.2 and 73.3). 

Older students, and those from disadvantaged schools and families, used the scholarship at 
lower rates than others. 

The study examined characteristics of the students who were offered but did not take advantage of the scholarship. 

These “non-users” or “decliners” were more likely to come from families that faced more significant challenges. 

• Although the SOAR Act identifies eligible applicants from low-performing SINI schools as a priority group for 
scholarships, a smaller proportion of these students took up this opportunity when offered to them (65 percent) 
than did applicants from non-SINI public schools (73 percent) or those already attending private schools 

(90 percent). It is important to note that 72 percent of students from private schools who were awarded schol- 
arships were already attending a participating private school when they applied to the OSP; not having to move 
schools may account for the high proportion of those applicants who used their OSP scholarship (figure 4-3). 

• Just over half (5 1 percent) of high school students awarded scholarships used them the next year, compared to 
78 percent and 69 percent for elementary and middle school students, respectively) (figure 4-4). 

• Students were less likely to enroll in a private school with an OSP scholarship if their parents were unemployed 
or not working full- time (69 percent) than if at least one parent was working full time (79 percent). A lower 
share of applicants living with an unmarried parent used their scholarships (70 percent) than those with married 
parents (79 percent). The educational levels of parents and guardians does not seem to be a major factor, since 
the use of scholarships is comparable for applicants with parents who did and did not attend college for some 
period of time (figure 4-5). 


37 See Howell, Wolf et al. for more information on scholarship use rates in programs other than the OSP 
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Figure 4-4 Percentage of Applicants Offered Scholarships Who Use Them by Students’ Grade Level at the 
Time of Application, 2011-12 and 2012-13 Combined 



Grade Level at the Time of Application 


NOTE: N = 1,014 

SOURCES: School-related data are from the Trust’s OSP application database and lottery files, and the Common Core of Data (http://nces. 
ed.gov/ccd/) ; data about scholarship use are from the Trust’s OSP Payment files. Years refer to school years. 


Figure 4-5 Percentage of Applicants Offered Scholarships Who Use Them by Parent Characteristics, 
2011-12 and 2012-13 Combined 
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NOTE: N = 1,014. Data were missing on college for 6 parents, on employment for 2 parents and on marriage for 1 parent. 
SOURCES: Data are from the Trust’s: OSP application database, lottery files, and the OSP Payment files. Years refer to school years. 
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Appendix A 

Determining the Number of Students (Sample Size) 

Needed for the OSP Evaluation 

A rigorous evaluation of the DC Opportunity Scholarship Program must ensure that program impacts of a policy 
relevant size can be estimated reliably and with adequate precision. Precision, in turn, depends on the number of 
participants in the evaluation, both students assigned to the scholarship (treatment) group and those assigned to 
the no-scholarship (control) group by the lottery. The larger the sample of students in the evaluation, the more 
reliable the impact estimates will be and the smaller the size of the impact that can be estimated with confidence. 
This same concept applies in political surveys, where the “margin of error” is typically larger when fewer individ- 
uals are polled. 

Using statistical information from the prior evaluation of the OSP, we have calculated the “minimum detectable 
effect size” likely to be achieved with various numbers of study participants (table A.l). Because effect sizes are 
in a unit that is not readily interpretable, we translated them into percentile point differences between the treat- 
ment and control groups (impacts) on the evaluations achievement assessment (the Terra Nova), and percentage 
point differences for other outcomes (such as percentage of students achieving proficiency). An important row to 
examine is that for 1,200 students, since the number of students who participated in lotteries in spring 2012 and 
spring 2013 is 1,255. Were the evaluation to base its impact estimates on this current sample, it could detect an 
impact of about 5 percentile points (a minimum detectable effect size of 0.128). If the program leads to the same 
effect size for test scores as the prior OSP evaluation found (0.1 1), the study needs at least 1,600 students in the 
sample for that effect size to be statistically significant. Our aim is to include approximately 1,800 students in the 
new evaluation, to be able to reliably estimate impacts for key subgroups ( e.g ;, students from SINI schools). 
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Table A-l Minimum Detectable Effect Sizes for Test Scores and Other Outcomes 

Total 

Minimum Detectable 

Equivalent 

Minimum Detectable 

Equivalent 

Number of 

Effect Size 

Percentile Point 

Effect Size 

Percentage Point 

Students 

(Test Scores) 

Difference 

(Other Outcomes) 

Difference 

100 

0.198 

7.9 

0.251 

12.6 

200 

0.181 

7.2 

0.229 

11.5 

300 

0.168 

6.7 

0.212 

10.6 

400 

0.157 

6.2 

0.198 

9.9 


500 

0.148 

5.9 

0.187 

9.3 

600 

0.140 

5.6 

0.177 

8.9 

700 

0.134 

5.3 

0.169 

8.5 

800 

0.128 

5.1 

0.162 

8.1 


900 

0.123 

4.9 

0.156 

7.8 

1000 

0.118 

4.7 

0.150 

7.5 

1100 

0.114 

4.6 

0.145 

7.2 

1200 

0.111 

4.4 

0.140 

7.0 


1300 

0.107 

4.3 

0.136 

6.8 

1400 

0.104 

4.2 

0.132 

6.6 

1500 

0.102 

4.0 

0.129 

6.4 

1600 

0.099 

3.9 

0.125 

6.3 


1700 

0.315 

12.4 

0.398 

19.9 

1800 

0.257 

10.1 

0.325 

16.2 

1900 

0.257 

10.1 

0.325 

16.2 

2000 

0.222 

8.8 

0.281 

14.0 


NOTE: Calculations of minimum detectable effect sizes assume that the t-test is two-sided with an alpha level of 0.05, a power level of 0.80, covariates 
explain 50 percent of test score variance and 20 percent of other outcome variance, and that the treatment group is 50 percent of the sample and the 
response rate for follow-up data is 80 percent. 

Calculations of equivalent percentile point differences for test scores and percentage point differences for other outcomes assume that the starting point 
is the median, 50 percent, for both. For example, a minimum detectable effect size of 0.128 for a sample of 1,200 is equivalent to a student at the 50th 
percentile moving to the 55.1 percentile. 


26 












EVALUATION OF THE DC OPPORTUNITY SCHOLARSHIP PROGRAM 

An Early Look at Applicants and Participating Schools Under the SOAR Act 


Appendix B 

Planned Data Collection and Report Schedule 


Table B-l Completed and Planned Data Collection 


Type of Data Collection 


Data Collection 
Period 

Application 

Forms 1 

Testing 

Surveys 

Trust 

Records 2 

School 

Administrative 

Records 

Spring/summer 2011 

2011 applicants 

NA 

Pre-program for 
2011 applicants 

2011 applicants 

NA 

Spring/summer 2012 

2012 applicants 

Pre-program for 
2012 applicants 

Pre-program for 
2012 applicants 

2011 and 2012 
applicants 

NA 



End of year for 
2012 applicants 

End of year for 
2012 applicants 



Spring/summer 2013 

2013 applicants 

Pre-program for 
2013 applicants 

Pre-program for 
2013 applicants 

2011-13 

applicants 

2012 applicants 

Spring/summer 2014 

2014 applicants 

Pre-program for 
2014 applicants 

Pre-program for 
2014 applicants 

2011-14 

applicants 

2012 and 2013 
applicants 



End of year for 
2012 and 2013 
applicants 

End of year for 
2012 and 2013 
applicants 



Spring/summer 2015 

2015 applicants 

End of year for 
2013 and 2014 
applicants 

Pre-program for 
2015 applicants 

End of year for 
2013 and 2014 
applicants 

2011-15 

applicants 

2013 and 2014 
applicants 

Spring/summer 2016 

2016 applicants 

End of year for 
2014 applicants 

Pre-program for 
2016 applicants 

2011-16 

applicants 

2014 applicants 




End of year for 
2014 applicants 




1 Application forms were administered by and data entered by the Trust. 

2 Trust records include payment files and private school directories. 
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Table B-2 Planned Report Schedule 


Report Title 

Brief Description 

Expected Publication 

Evaluation of the DC Opportunity 
Scholarship Program: An Early Look at 
Applicants and Participating Schools 

Summary of implementation; focus 
on describing participating private 
schools, eligible applicants, and 
scholarship users 

Spring/summer 2014 

OSP Implementation Under the SOAR Act: 
2011-13 Applicants 

Summary of implementation; focus 
on the reasons parents choose their 
child’s school 

Spring/summer 2014 

OSP Implementation Under the SOAR Act: 
2011-14 Applicants 

Summary of implementation; focus to 
be determined 

Spring/summer 2015 

Evaluation of the OSP Under the SOAR Act: 
Impacts 1 Year After the Lottery 

Effects of the OSF? after 1 year, on 
student achievement in reading and 
math, parent and student satisfaction, 
parent and student perceptions of 
safety, parent involvement 

Spring/summer 2016 

Evaluation of the OSP Under the SOAR Act: 
Impacts 2 Years After the Lottery 

Effects of the OSF? after 2 years, on 
student achievement in reading and 
math, parent and student satisfaction, 
parent and student perceptions of 
safety, parent involvement 

Spring/summer 2017 
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Appendix C 

Weighted Participating School Data 


Figure C-l Characteristics of Participating Private Schools Under the DC School Choice Incentive Act 
and the SOAR Act, 2005-06 and 2011-12, Weighted by OSP Students Served 
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Percent Serving Percent Reporting Average Percent Percent Religiously Percent Catholic 
Grades 9-12 Tuition Rates Minority Students Affiliated 


Above OSP Cap Served* 


School Characteristic 


Incentive Act 2005-06 


SOAR Act 2011-12 


♦“Average Percent Minority Students Served” is calculated as the mean of the percent of each school’s student body that is from a minority 
racial-ethnic group (non-White). 

NOTE: The 2011-12 school year (under the SOAR Act) and the 2005-06 school year (under the DC School Choice Incentive Act) are used to 
compare the characteristics of the program under the two statutes, with these specific years chosen because: (1) both time periods were soon 
after each statute was passed and when significant funds were available for new scholarships, and (2) comprehensive data from the earlier 
evaluation were available for the 2005-06 year that were not available for other years. Tuition data for participating schools in 2005-06 are 
available for 60 of the 68 schools and for all 53 participating schools in 2011-12. Years refer to school years. When the proportion of schools is 
not weighted by the number of OSP students served (/'.e., each school is treated equally), the gaps between schools participating in the two time 
periods are larger (see figure 2-4). 

SOURCES: Data for 2005-06 come from Evaluation of the DC Opportunity Scholarship Program: Second Year Report on Participation (NCEE 
2006-4003); 2011-12 data are from the Trust’s OSP Participating School Directory and private school websites. 
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Figure C-2 Percentage of Private Schools With Published Tuition Above and Below the Voucher Amount for 
Schools Participating Under the SOAR Act, 2011-12, Weighted by OSP Students Served 
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Total 

(N = 1,633) 


NOTE: Schools were included in the category corresponding to the highest grade served. Of the 19 schools serving students in grades 9-12, 13 
also served K-8 students. Years refer to school years. When the proportion of schools is not weighted by the number of OSP students served 
(i.e., school were treated equally), the share charging above the voucher amount is much larger but similar for both K-8 and 9-12 schools (see 
figure 2-5). 

SOURCE: Data are from the Trust’s 2011-12 OSP Participating School Directory. 


Table C-l Characteristics of Participating DC Private Schools, 

2011-12, Weighted by OSP Students Served 

Percent serving high school 

35.2 

Average pupil-teacher ratio 

10.3 

Average years in operation 

75.0 

Average student population 

217 

Percent minority students 

90.0 

Percent faith-based 

81.1 

Percent Catholic 

51.0 

Percent non-faith-based 

18.9 

Number of schools overall 

53 


*NOTE: Data were available for all 53 participating schools. Years refer to school years. When school characteristics are not weighted by the number of OSP 
students served {i.e., school were treated equally), the average percent of the student body who are from minority-racial-ethnic groups and the share that are 
faith based (included Catholic) are all smaller (see first column of table 2-1). 

SOURCES: Private school data are from the Trust’s OSP Participating School Directory, private school websites and the Private School Universe Survey http:// 
nces.ed.gov/surveys/ pss/pssdata.asp . 
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